This invention relates to feedback affecting a computer display device with regards to motion of a pointing device interfaces to that computer.
The prior art teaches for methodologies interactions between a display device 10 and a pointing device 12 (see FIG. 1). There has been a continual evolution of content and interface regarding these two components resulting in mechanisms including windowing systems. Such windowing systems are often composed of a region (window) of the display device, further composed of sub-regions. These sub-regions include, by way of illustration, buttons, pull-down menus and icons activating specific window functions. These window functions often include but are not limited to opening and saving files, numeric and operator buttons on calculator and spreadsheet applications, copying, cutting and pasting highlighted material within the window to a clipboard.
Human physiology is inherently imprecise, as evidenced by an examination of raw sensor data from a mouse. This data is usually very xe2x80x9craggedxe2x80x9d. Operating systems and in particular, pointing device drivers, including mouse drivers, often possess built-in acceleration curves to smooth out these discontinuities on a global scale.
There are a number of situations regarding windowing systems in general when there is a need to reduce background mechanical noise with regards to pointing device movement. Illustrative examples include users with limited coordination, such as the very young as well as people possessing limited mechanical control of the pointing device. Such limited mechanical control can be additionally caused by background noise, such as using a notebook computer on a bumpy airplane or transit ride. Existing interfaces lack sufficient noise reduction support, causing users to lose productivity in such circumstances.
A further limitation to existing interfaces is a lack of readily accessible control of pointers as they approach the boundary of a display region such as a window. In a number of applications including multimedia players, as the user moves the pointing device further and further in a given direction, the action should continue, even if it is at the boundary of the display region. The application developer should have control of what happens in such situations, rather than be governed by decisions made by the operating system designers. Form should best follow function, allowing the application developer the freedom to express the response of the pointer with regards to the display system in a manner consistent with the interaction with the user""s activity with the pointing device.
Other evolutionary results include display mechanisms for 2-D, 3-D and 4-D systems utilizing one or more specific perspectives, views or projections onto specific planes. Such multidimensional displays often map from a virtual region containing sub-regions and representing the pointing device as having an associated virtual pointer location relative to the virtual region. Such pointers may or may not be visible on the display device. When visible, they often appear as a cursor. There is a need in many of these systems to alter the overall motion of the pointer based upon some or all of the sub-regions the pointer is traversing. Such alterations include but are not limited to the portrayal of friction, currents, attraction and repulsion, impulsive and inertial motion effects induced on the pointer by proximity to one or more traversed sub-regions, and thus affecting the display.
In 2-D and 3-D system display mechanisms, there is an option of using direct force feedback pointing devices, which has prior art, particularly in the 2-D situation.
Such pointing devices tend to cost more than standard pointing devices and also to be subject to the same problems of background mechanical noise as discussed above.
The mechanical complexity, and consequently, the cost for 3-D implementations is significantly greater than for 2-D implementations. There are no known direct feedback mechanisms for higher than 3-D system interfaces.
Prior art system interfaces incorporating a pointing device and display often further incorporate a user interface to calibrate the pointing device. On a number of systems including Windows systems, such user calibration tends to be known as control panels. A correlation factor between displaced movement of the pointing device and resulting movement of a displayed cursor is a common calibration control in such interfaces. This is a worthwhile basic user interface control, but it does not permit for the calibration of friction or traction based upon traversing various sub-regions, nor any form of attraction nor inertia.
Further, some prior art interfaces have shown a slowing effect when traversing a specific sub-region. This has most notably been seen in a slowing when crossing the xe2x80x9cOKxe2x80x9d button and xe2x80x9cCancelxe2x80x9d button on certain prompt windows in the Microsoft Windows operating systems. While this is an improvement, it does not tend to filter out background mechanical noise as discussed above. It is further not available to developers to control for other graphical interface elements, such as other kinds of controls, pull down menus, etc.
Adobe Illustrator provides a limited snap to handle capability when moving one object with handles near a second object with handles. If the handles come within a small number of pixels of each other, say four pixels, the neighboring handles of these objects snap to each other, with the one in motion moving toward the other object. To separate them requires moving the cursor greater than four pixels to again separate them. While there are objects with can attract cursor driven objects, it is strictly a snap operation. There are no control panels or other mechanisms by which this can be extended or modified. There is a fixed distance at which it occurs. While useful, it is not very flexible and possesses no obvious way to extend such a mechanism to induce motion in any other form.
FIG. 2 portrays a system block diagram of prior related art showing the interface of a microprocessor 20, display 10, a pointing device 12 and other circuitry including but not limited to mass storage device(s) 14 and network interface 16. In such typical prior art systems an IO bus 18 often provides the communication for such devices. Communication 18 includes but is not limited to treating such circuitry as addressable devices on an ISA bus or Universal Serial Bus (USB). In certain examples implementation of communication 18 employs more than one such bus mechanism.
Display devices 10 and pointing devices 12 are not the subject of the invention. Prior art pointing devices 12 for desktop systems with electrically interfaces include but are not limited to mouse pointing devices, pen tablets, touch pads and joy sticks. Smaller computers such as notebook computers and Personal Digital Assistants often incorporate touch pads, pointing buttons, and pen tablet pointing devices. Other prior art pointing devices 12 incorporating a wireless interface include but are not limited to 3-D mice and selector controls for various television, television-like and entertainment systems. The most frequent display devices are CRT displays and flat panel displays, some of which are in color and some of which are in black and white, sometimes further capable of shades of gray and one or more speakers. Personal Digital Assistants often integrate the display and a pen tablet into a veiwable writing surface.
Optionally, mass storage devices 14 commonly include but are not limited to removable and non-removable media devices. Non-removable media include but are not limited to hard disk drives and non-volatile solid state disks. Removable media include but are not limited to floppy disk drives, CD ROM players, PCMCIA flash disk drives and Compact Flash drives.
Optional network interfaces 16 available in typical prior art systems include but are not limited to wireless and wireline communication schemes. Wireless schemes include but are not limited to InfraRed (IR) ports, wireless packet modems used in CDPD, IS-95 and GSM radio communications schemes and W-CDMA. Wireline interfaces include but are not limited to various forms of LAN interfaces, wireline modems and various packet transport schemes including but not limited to ATM. LAN interface protocols include but are not limited to Ethernet, Fast Ethernet, Gigabit Ethernet and Fiber Channel. Wireline modem interfaces protocols include but are not limited to 33.3K, 56K, ADSL and ISDN. Any of the above protocol interfaces may additionally support Wide Area Network (WAN) protocols such as TCPIP and the Internet, with its World Wide Web.
Microprocessors 20 typically require local memory, which includes RAM 24 and usually some form of non-volatile memory 26. Nonvolatile memory includes but is not limited to one or more of the following: ROM, Flash Memory, EPROM and writeable control stores. Such memory, particularly in hand held computers such as PDAs incorporates most if not all of the operating system and applications within the device and may include a file management system using some or all of the non-volatile memory as an emulated disk drive. In certain prior art systems display 10 may be further coupled to RAM 24 by an additional bus to support high speed data transfers between these units.
Evolutionary systems applications such as Windowing systems and multi-dimensional viewers as described above often run under a real-time, event driven operating system including but not limited to MacOS, various versions of Microsoft Windows (3.1, 95, 98, NT, CE) and PalmOS, to name some examples. Several applications have been built in such event driven systems using standard procedural languages such as C and assemblers. Other applications operate in an object-oriented environment. The object systems are often developed in programming language environments including but not limited to C++, Java and Visual Basic.
Other approaches include but are not limited to Lingo by Macromedia, which while not explicitly revealing the event queue, triggers actions based upon semantic constructs such as xe2x80x9con MouseUpxe2x80x9d and xe2x80x9con MouseDownxe2x80x9d. Such approaches implicitly use the standard approach discussed above where the event queue is processed in terms of exactly and only the top element of the event queue. The generated application has added control to exactly those event handlers which are referenced in the source code in the exact context specified by that source code.
FIG. 3 depicts the operation of a representative event queue in such prior art systems. The word queue in this context refers to a First In First Out ordered list. An event queue""s list elements are known as events. Each event is further composed of an event type 40, or event handle. An event activity 42, such as button up or down may be further associated with an event in some operating systems. Such event activities may be incorporated into the event type or handle, or may represent a separate field within the event. Each event in an event queue possesses a link to the next most recent event in the queue 44. By way of illustration, an event queue will be shown in two states, 38a and 38b, demonstrating the effect of processing an event in a typical prior art software application.
Event queue 38a contains an oldest event 30, which has a component event type or handle 40 and a link 44 to the next oldest event 32. The next oldest event 32 has a component event type or handle 40 and a link 44 to next oldest event 34 from event 32. This event links to a later event, etc. until a most recent event 36 links to a nil pointer, signifying that it is the most recent event in the queue.
Event queue 38b contains an oldest event 32, which has a component event type or handle 40 and a link 44 to next oldest event 34. This event links to a later event, etc. until a most recent event 36 links to a nil pointer, signifying that it is the most recent event in the queue. The event 30 has been removed from the queue.
FIG. 4 depicts a typical flowchart for an event handler of the prior art. Such event handlers are often implemented as threads in a real-time operating system. A thread usually entails a return stack and register state context of the CPU in a native code execution environment. JAVA and other portable interpretive tools will allocate and save the interpreter control state context for a thread. In both native code and interpretive situations, there is an instruction or program pointer which points to the next (or sometimes an offset of the current) instruction location being executed.
The first procedural activity encountered after setting up the primitive thread environment is the initialization 50 of the event handler thread. Once initialized, the next event in the event queue is retrieved 52. This operation makes it possible for the thread to examine the event and at the same time removes it from the event queue. Referring back to FIG. 3, situation 38a illustrates the condition of an event queue before retrieving the next event and situation 38b illustrates the event queue after retrieving the next event. Process event 54 interrogates the event to determine what needs to be done, usually by considering its event type 40 and event activity 42, and then performs one or more functions using the contents of the event and other information available to the thread. The state of the thread and/or retrieved event is then used to decide whether to quit the application 56. If the decision is to quit the application, then usually there is an operation 58, which cleans up residual memory among other things, and then causes the application thread to exit or terminate. If the decision is to continue the application, then a branch 60 back to retrieve the next event occurs, and a loop is effected. The branch 60 may also entail a time delay or wait function, which often causes the thread to be stored until the next event is found or a given minimal period of time has elapsed.
While there are numerous operating systems which today support real-time and windowing primitives with significant divergence between them, the basic mechanism described here reveals some very uniform semantic constructs: The event handlers consistently retrieve just one event, the oldest in the ordering of the event queue (52). They process one event at a time (54). If the decision to stop the thread is not made, at some possibly later time, they loop back (60) to retrieve the next event and process it. This same mechanism is observed and recommended consistently for applications in all the discussed operating systems (MacOS, various versions of Microsoft Windows (3.1, 95, 98, NT, CE) and PalmOS) and runtime environments mentioned above (C, C++, Java and Visual Basic). Other development approaches like Lingo implicitly utilize this recommended method.
While this mechanism has significantly contributed to the great advances of our time, it poses a significant set of restrictions upon user interfaces. Motion is an effect of change and accumulation, looking at one event cannot tell enough to know or understand a flow and calculate motion. Such mechanisms are often conceptually embodied as a finite digital filter acting upon a collection of near past and/or near future events relating directly to the object in motion, such as a pointing device and its virtual region associate, the pointer. Altering a pointing device location while that data is in an event queue within a windowing system is extremely difficult often leading to applications of questionable reliability. What is needed to achieve such things is a reliable and effective mechanism for altering the contents of an event queue regarding events associated with a pointing device.
Many situations place significant requirements regarding modeling motion upon traversed sub-regions. The portrayal of friction, currents, attraction and repulsion, impulsive and inertial motion effects induced on the pointer is done against the backdrop of the discrete nature of digital computers. Motion effects such as inertia, velocity and acceleration are often best formulated as finite difference schemes. Event queues do not work well in tracking such things. Event queues usually do not report cursor positions at regular time intervals, only when buttons are pushed, etc. Further, event queues do not possess a consistent number of entries for pointer locations at discrete times with predetermined time intervals in the near past and near future of the location being modified, which is a distinct disadvantage in calculating finite difference schemes.
While physics has for centuries provided our basic conceptual paradigm of motion, simulation environments allow for and in fact facilitate a much broader sense of motion than the physical world. Consider the inverse square laws of gravitation and electromagnetism. Such formulations, while well confirmed by experiment, do not lend themselves well to use in human-computer interactions. Further, while the Lorentz force law of electromagnetism and others laws in other branches of physics give us direction dependent induced motion, computationally, these are a small subset of the range of expression a computer can provide to a virtual environment designer.
Inference rules are an established tool in the construction of artificial intelligence systems, in particular, expert systems and constraint based systems. Rules are generically considered to provide a conditional and an action, similar to an if-then construct as found in a procedural language such as C or FORTRAN. There is one fundamental difference between rules in general and the if-then construct of procedural languages, the ordering of rules tends to be irrelevant, whereas in procedural languages and some logic programming languages such as Prolog, the exact sequence of statements is held essential to the meaning of the statements. When the exact sequencing of rules is essential, semantic restrictions forced on applications so developed are often unnatural and sometimes debilitating. Example: when the collection of rules is derived from a database possessing no inherent natural ordering of objects or rules, what is optimal often entails treating all relevant rules on an equal footing. Constraint based systems generally involve a collection of variables which are to satisfy a collection of rules, often known as constraints.
Rule based development systems are available from a number of sources, including implementations of concurrent Prolog and CLIPs, by way of example. Such systems have been used to advantage in generating rules systems including but not limited to several applications targeting specified semiconductor process technologies for VLSI layout verification and generation software, as well as tools interrogating a database for data patterns satisfying specific constraints or inference rules. However, such tools have not been seen as providing an effective, intuitive user interface to specify induced motion based upon traversing one or more geometric objects, or portions of such objects. Such tools also do not provide a mechanism for intercepting the pointing device event stream and controlling the pointer or cursor independently of the default mechanisms provided by the operating system upon which such tools reside.
The last few years have seen the emergence of intelligent agents as a class of software entities with at least limited capability to act independently of a user. These agents traverse the Internet, or possibly one or more Intranets or Extranets, accumulating information which is then presented back to its user at some predetermined time. While these entities are still in their infancy, one substantial issue is the configuring of these agents toward or away from various sites or objects in a virtual space. For instance, a user might want to know about fishing nets, not neural nets.
Many users prefer geometrically oriented interfaces where proximity of two or more entities along a dimension indicates a closeness of context or relevance based upon one or more selected criteria. There is no known user interface supporting the presentation of such agent activity or configuring agents based around a multi-dimensional paradigm with the ability to guide the motion of the agents and/or user in a fashion comparable to the physical paradigms of motion.
One aspect of the present invention is implemented as a cursor control system including a display having a plurality of distinct regions, a pointing device influencing a pointer on the display and a mapper. The mapper is responsive to the pointing device and to a position of the pointer relative to the distinct regions, and operative to affect a continued movement of the pointer. The continued movement is a transform of the pointer in at least one of the regions.
This is advantageous in making the perceived cursor motion an avenue of feedback as to the properties of regions the cursor traverses providing a virtual force feedback mechanism. This virtual force feedback provides an inexpensive mechanism for motion altering effects in 2-D virtual environments. This virtual force feedback mechanism is even more useful in 3-D virtual environments, where there are very few direct force feedback alternatives and these alternatives cost a great deal more than similar approaches to 2-D direct force feedback. This virtual force feedback mechanism has no direct force alternative in 4-D and higher dimensional virtual environments.
This method is also advantageous in reducing background mechanical noise by an arrangement of regions and their continued pointer motion transforms. Such advantages can significantly improve the productivity of people. In addition, it can be used to aid those with significant mechanical disabilities as well as people using computers in mechanically noisy environments, such as on bumpy airplanes as well as crowded trains and buses.
Another aspect of the present invention includes a method for controlling a pointer in an object oriented graphical environment including the operations of: developing a list of objects pertaining to graphical regions in a specified area of a display, determining for all of the objects which are pertinent to the pointer location and modifying a movement on the display based upon properties of the pertinent objects.
This method is advantageous in the development of interactive environments where the pointer motion is dependent upon some or all of the pertaining objects in a graphical region. Such lists of objects may be statically determine during application development, developed while the application is running, possibly being stored and retrieved to support an evolutionary object motion library. Such object lists may be stimulated by interaction with software agents.
A further method of the present invention for controlling a pointer in an operating system having an event queue includes: examining the event queue for original pointer events generated by a pointing device; removing the original pointer events from the event queue; and generating new pointer events based upon the original pointer events. Inserting the new pointer events into the event queue.
This is advantageous in permitting an efficient mechanism replacing and/or modifying an existing event handler. This new event handler permits interaction in a transparent fashion within a real-time operating environment employing event queues and windows style pointers based upon the events associated with a pointing device by removing original pointer events and generating new pointer events based upon these original pointer events. Such real-time environments include but are not limited to operating systems (MacOS, various versions of Microsoft Windows (3.1, 95, 98, NT, CE) and PalmOS) and runtime environments mentioned above (C, C++, Java, Visual Basic and Lingo).
It is further advantageous in supporting the examination of the entire event queue for original pointer events, rather than just the top of the event queue. This increases the reliability of the mechanism, given the wide diversity of potentially active applications and the interaction of all their event handlers.
A further method in accordance with the present invention for controlling a pointer in an operating system having an event queue includes; removing from an event queue original pointer events generated by a pointing device; generating new pointer coordinates based upon the original pointer events; and displaying the pointer at the new pointer coordinates.
This is advantageous in permitting an efficient mechanism replacing and/or modifying an existing event handler. This method permits interaction within a real-time operating environment employing event queues and windows style pointers based upon the events associated with a pointing device. Such real-time environments include but are not limited to operating systems (MacOS, various versions of Microsoft Windows (3.1, 95, 98, NT, CE) and PalmOS) and runtime environments mentioned above (C, C++, Java, Visual Basic and Lingo). The generation of new pointer coordinates for a pointer event provides a mechanism where induced motion effects may affect the pointer location at the event handler level just before the pointer is displayed.
A further method of the present invention for controlling a pointer on a display which includes: manipulating a pointing device to move a pointer on a display; observing the pointer motion behavior on the display relative to one or more objects on the display; and modifying the manipulation of the pointing device upon observed behavior to continue the movement of the pointer.
This is advantageous in providing the user/observer a new paradigm of interaction with complex entities, where the observed motion communicates content or attributes of content of objects on the display to the user. Note that the user may be controlling the pointing device. Note that a software or algorithmic agent may be controlling the pointing device, which could be a virtual pointer into a virtual space. Note further that a collection of software or algorithmic agents may be controlling a collection of pointing devices. Thus the modification of the manipulation of the pointing device may entail either modification of the manipulation of a physical pointing device or modification of the virtual manipulation of a virtual pointer in a virtual space.
This method may be advantageously performed by a software tool interrogating an existing program environment and determining which elements should be given induced motion rules and incorporating those induced motion rules into the application""s operating system environment.
Software agents may further advantageously use this method to organize the presentation of informational systems interrogations, deriving objects based upon criteria or rules specified by either a higher level, software or algorithmic controlling agent or a user. Such informational systems include but are not limited to database systems, one or more Intranets, one or more Extranets and/or the Internet.
A method of the present invention of interfacing a pointing device and a display device with respect to a virtual region of virtual locations includes; maintaining at least one virtual sub-region with at least one virtual location and at least one induced motion rule; associating the pointing device with a virtual location of the virtual region; forming a list of the sub-regions containing the virtual location of the pointing device; and modifying the virtual location of the pointing device by applying the induced motion rules of the list of the sub-regions containing the virtual location of the pointing device.
This interface method is advantageous by providing one or more sub-regions with induced motion rules, which can alter the virtual location of a pointing device. The collection of sub-regions sharing the pointing device location can then interact as a rule based system to affect the location alteration. This is particularly advantageous in situations where the virtual region is of higher spatial dimension than 2, say 3, 4 or higher dimensions. This is further advantageous when the induced motion rules are not analogues of the standard physical rules of motion, such as when direction of repulsion may be determined by the direction best satisfying some search criteria.
These and other advantages of the present invention will become apparent upon reading the following detailed descriptions and studying the various figures of the drawings.